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ABSTRACT 


In this thesis, we present a system for annotating persuasion in conversation based 
on a social-psychological model. We augmented the social model developed by James 
Cialdini with some of our own categories for annotators to label. The conversations 
consisted of 37 hostage negotiation transcripts from private and public sources, with all 
personal information removed from the private source transcripts. We evaluated the level 


of agreement between annotators using Cohen’s Kappa measurement. 


Our initial results showed only fair to moderate agreement, with an average kappa 
score around 0.41 for transcripts of significant length (over 200 utterances). Based on 
these results, annotators revised the annotation model and eliminated some categories of 
persuasion while adding “other” as a catchall for any persuasive utterance not covered by 
Cialdini. The revised model showed a significant increase in agreement with an average 
kappa score of 0.78 for transcripts of significant length (over 200 utterances). Based on 
this revised model, annotators adjudicated a final persuasion corpus for the 37 transcripts 


that will be used in future works on persuasion detection. 
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EXECUTIVE SUMMARY 


In this thesis, we present a system for annotating persuasion in conversation based 
on a social-psychological model. The conversations consisted of 37 hostage negotiation 
transcripts from private and public sources, with all personal information removed from 
the private source transcripts. Our researcher used the principles outlined in James 
Cialdini’s work on persuasion. Cialdini’s principles of persuasion are reciprocity, 
commitment and consistency, liking, scarcity, authority and social proof. To these six 
categories we also added “reason.” Using this model and a codebook outlining 
persuasive examples in hostage negotiations, we assigned different transcripts to each 
annotator (with some overlap to test for agreement) and had the annotators label each 
utterance as persuasive or non-persuasive based on the model. We evaluated the level of 
agreement between annotators using Cohen’s Kappa score and the overlapping 


transcripts. 


Our initial results showed only fair to moderate agreement, with an average kappa 
score around 0.41 for transcripts of significant length (over 200 utterances). Based on 
these results, annotators revised the annotation model and eliminated the category of 
“reason” while adding the category of “other” as a catchall for any persuasive utterance 
not covered by Cialdini. In addition, the codebook was updated to include more 
examples and more specific guidelines for recognizing persuasive utterances. The 
revised model showed a significant increase in agreement with an average kappa score of 
0.78 for transcripts of significant length (over 200 utterances). Based on this revised 
model, annotators adjudicated a final persuasion corpus for the 37 transcripts that will be 


used in future works on persuasion detection. 
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I. INTRODUCTION 


A. COMPUTER SCIENCE AND PERSUASION: THE GOAL 


The goal of this research was to develop an annotated corpus of persuasive 
conversations and is the first step in developing machine learning systems that can 
automatically detect persuasion in conversations. This corpus was developed from 37 
hostage negotiations because they have numerous examples of persuasion. Both the 
negotiator and the hostage taker have goals they wish to fulfill, and these conversations 
provide annotators with many instances of persuasion attempts for training and 
experimentation. The goal of the annotators was to agree consistently on the persuasion 
attempts, from either party, in a transcript, and to use these annotated transcripts as a 


corpus for future work. 


B. MOTIVATION 


The motivation for this research came from several sources. One source was that 
this field of research in computer science was largely unexplored. There has been 
research in related areas like perspective analysis, but persuasion analysis remains 
untouched. Another source of motivation was the power that persuasion has over 
individuals. An example of this power can be seen in suicide bombers. While there are 
many factors that go into a suicide bomb operation, convincing a person to strap 
explosives to their body and blow themselves up at a target requires some form of 
persuasion. Based on this research, it may be possible to construct a machine learning 
system that can automatically detect persuasion in conversations, like those on foreign- 


fighter recruitment Web sites, and alert authorities or analysts to the situation. 


C: DEFINING PERSUASION AND THE ANNOTATIVE MODEL 


There are many questions to be answered when dealing with persuasion in 


conversation. What is persuasion? Is the detection of persuasion in conversation innate 


to people or are there specific types of markers for persuasion attempts? If persuasion 
detection has specific markers, can these markers be learned and identified by 
annotators? Our research attempted to answer all of these questions by annotating a 
corpus of 37 hostage negotiations with persuasion tags based on the social-psychological 


model of James Cialdini. 


In general terms, persuasion is the ability of one party to convince another party 
to act or believe in some desired way. When defining a persuasion attempt in a 
conversation corpus, it is simply the agreement between two or more annotators that an 
utterance is persuasive based on a social model. In other words, if annotators can agree 
that some utterance of a conversation is meant to be persuasive, then it is. The key for 
annotators is to use a social model that allows them to have consistent agreement. For 
our research, we used the model defined by James Cialdini with six categories of 
persuasion: reciprocity, commitment and consistency, liking, scarcity, authority and 
social proof. In addition, we added “reason,” and later “other,” as persuasion categories. 
Annotators were asked to tag utterances in transcripts as persuasive or non-persuasive 


based on these categories. 


D. RESULTS 


The results from the initial experiment showed fair agreement but it was not very 
convincing. Based on these lackluster results, the social model was revised, eliminating 
some categories and adding others, and the experiment was rerun. Based on the revised 
model, the annotators showed substantial agreement with significant improvement from 
the original model. Using the revised model, a final adjudicated corpus of hostage 


negotiations was created between the annotators. 


E. ORGANIZATION OF THESIS 


This thesis is organized into several chapters: background information, the 
codebook and experiment design, results and analysis, and the appendix. The 
background information outlines related work in computer science (specifically 


perspective analysis), the social model of persuasion developed by Cialdini, and the 


2 


agreement measurement calculations. The codebook and experiment design sections 
details specific examples of each type of persuasion category and describes the 
annotation scheme. It also describes the revised model with a more specific codebook 
used by the annotators when the experiment was rerun. The result and analysis section 
details the agreement between the annotators and also notes any other significant 
findings. The Appendix is an example of a fully tagged transcript for reference to the 


reader. 


THIS PAGE INTENTIONALLY LEFT BLANK 


I. BACKGROUND 


A. INTRODUCTION 


There has not been a great deal of computer-science research in persuasion 
detection. As far as we know, this research is the first of its kind to develop an 
annotation scheme that would allow for computational modeling of persuasion. As such, 
we are basing our annotation scheme on the most prevalent social-psychological theory 
of persuasion. This work was done by James Cialdini and is the foundation of most work 
in persuasion. Our research essentially codifies Cialdini’s work and adds our new 
category, reason. This chapter will briefly explore related work, specifically perspective 
analysis, and then explain the social-psychological model that will be used in the 
annotation scheme. An evaluation method will also be explained to assess how much 


agreement exists between annotators. 
B. PERSPECTIVE VERSUS PERSUASION 


The closest research in computer-science to persuasion detection is the work on 
perspective detection, meaning the ability to detect the point of view from which a 
document is written. Though our research does not use perspective, the methods are 
important such as those developed by Lin, Wiebe and Hauptmann (Lin, Wilson, Wiebe, 
& Hauptmann, 2006). The researchers used the bitter-lemons corpus (bitterlemons.org) 
to test perspective classification using different types of machine-learning classifiers such 
as Naive Bayes and Support Vector Machines. The bitter-lemons corpus is a collection 
of documents that have the same subjects but are written from different perspectives 
(Israeli and Palestinian). By using an automatic subjective sentence classifier (Riloff & 
Wiebe, 2003) they split each document into subjective and non-subjective sentences. 
Using computational models, they were able to correctly identify the documents 
perspective, either Israeli or Palestinian, with a high degree of accuracy (over 97% for the 
editors of bitter-lemons and over 86% for guest writers). While they had a high degree of 


accuracy at the document level, they also tried applying their models to the sentence level 


5 


by attempting to classify a sentence with an Israeli or Palestinian perspective. A 
sentence’s perspective was labeled according to the document from which it was 
selected. Their accuracy classification at the sentence level was just over 75%, implying 
that perspective classification at the sentence level is much more difficult than at the 


document level. 


C. PERSUASION PRINCIPLES 


i Background 


The persuasion model for this research was based on the principles researched and 
developed by James Cialdini and used eight principles to mark persuasion in 


conversations. 


The study of persuasion involves the psychology of compliance, meaning the 
ability to get one party to agree to a set of conditions or desires of another party. 
According to Cialdini, the principles of persuasion transcend language, ethnicity, and 
cultures. These principles may be prioritized differently from culture to culture, but their 
existence and influence is important. This “transcendence” exists because persuasion 
principles are fixed action patterns. Thus the assumption of this body of research is that 
these principles are not learned, meaning they are innate and instinctual, but their 
ordering can be learned. Cialdini calls these patterns “click, whirr’ behaviors. “Click 
and the appropriate tape is activated; whirr and out roils a standard sequence of 
behaviors” (Cialdini, Influence: Science and Practice, 2001). The key to detecting 


persuasion is determining what exactly triggers the “click, whirr” response. 


Fixed action patterns are not limited to persuasion. They are actually more 
readily recognizable in nature. Courtship and mating rituals in the animal and insect 
kingdoms are examples of actions that precipitate pre-conditioned responses, as if the 
responses were stored on tapes in the subconscious. Cialdini documented eight principles 
of persuasion in his research that trigger compliant responses, and these principles will be 


used in annotating the police and hostage negotiation transcripts. 


In police and hostage negotiations, there are usually many persuasion attempts 
between the two parties (law enforcement and lawbreakers). This thesis will not only 
attempt to identify the persuasion attempts on the law enforcement side, but also the 
attempts from the lawbreakers. If demands are unreasonable, the police or Federal 
Bureau of Investigation (FBI) do not usually accommodate, but they may if the demands 
are small, such as food or a pack of cigarettes. Police often use small desires to make 
agreements (release hostages by a certain time, trade hostages for food, etc.) with the 
lawbreakers. There are occasions when the lawbreakers can persuade the law 
enforcement officers to acquiesce to small demands without giving any real concessions. 
Since this thesis is focused on any kind of persuasion attempt, an attempt from either 


party will be noted. 
2. Principles of Persuasion 


Cialdini identified six main principles of persuasion through social experiments 
and statistical analysis: reciprocity, commitment and consistency, social proof, liking, 
authority, and scarcity. In addition to these six elements, this research uses an additional 
one: reason. Cialdini mentioned reason, but it did not have its own category in his work. 
It will, however, be included as a category in our work, due to the nature of police and 
hostage negotiations where negotiators tend to give many reasons or justifications for 


their requests. 
a. Reason 


Reason in persuasion attempts is an explanation or justification for some 
kind of request. This justification is very common, and not just in police and hostage 
negotiations. For example, a mother might say to a child, “Do not touch the stove when 
it’s hot, because you will get burned.” This type of statement may lie in an action- 
consequence realm, but reasons alone can be influential, even if the reasons are 


meaningless. 


An experiment conducted in 1978 by social psychologist Ellen Langer 


showed the power of reason in invoking a compliant response (Langer, Blank, & 
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Chanowitz, 1978). The experiment utilized 120 subjects and Langer’s students tried to 
cut in front of the subjects at a photocopier with a minimum wait time of five minutes 
between requests. There were three separate cases for the subjects and Langer’s students. 
In one case (no information), the student simply asked to cut in line with the phrase, 
“Excuse me, I have five pages. May I use the Xerox machine?” This case had only a 60 
percent compliant response from those waiting in line. In the second case (sufficient 
information), the student asked, “Excuse me, I have five pages. May I use the Xerox 
machine because I am in a rush?” This instance had a 94 percent compliant rate. This 
increase may be somewhat intuitive and not very insightful. The people in line were 
given a reason (“I’m in a rush”) and they politely accommodated the request. The key 
result of this experiment lies in the third case (placebic information), where the student 
asked, “Excuse me, I have five pages. May I use the Xerox machine, because I have to 
make copies?” This case offers no new information from the first case (with 60 percent 
compliance), but the results were astonishing. Nearly 93 percent complied with the 


request. 


Table 1 shows the summation of their results (Langer, Blank, & 
Chanowitz, 1978). It is worth noting that if the favor was considered a large one, the 
results decreased. A favor was considered small if the number of pages the student 


wanted to copy was fewer than the subject’s number of pages to be copied and large 














otherwise. 
Table 1. | Results from Langer’s Experiment 
Reason 
Favor No information Sufficient Information Placebic Information 
Small 60% 94% 93% 
Large 24% 24% 24% 














The effect of giving a reason, even a meaningless one, is significant 
because it demonstrates the ability for reason to affect compliance outcomes. Cialdini 


notes that the word “because” triggered a “click/whirr” response. In_ hostage 
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negotiations, negotiators often give a reason for requesting a particular action, so this 
experiment will include reason as a category of persuasion, because purely lexical 


information can have a dramatic impact. 


Below is an example of using reason in a persuasion attempt. This excerpt 
is part of a transcript between a 911 dispatcher and a suicidal man (Grant) who had called 
the police and was threatening to kill himself. The transcript was provided by the 


American Civil Liberties Union (aclusandiego.org, 2008). 


911 DISPATCHER: Okay. Grant? 
GRANT: Yes. 


911 DISPATCHER: I’m here, talk to me. You just keep spilling your 
feelings to me, just to talk to me, just tell me what 
you need. 


GRANT: I don’t know what I need. 


911 DISPATCHER: If you have to cry, you cry. Okay? If it makes you 
feel better, I want you to cry, because sometimes 
crying helps, you know, makes you think better. 
Nobody is going to come in, okay, but I want you to 
talk to me so we can help you. This has been—this 
has to end. Because I don't want you to hurt yourself. 
But I need you to put down that gun. If you put it 
down, nobody is going to come in. We’re going to 
talk to you, okay? Can you put it down for me, away 
from you, because I don’t want anything to happen 


to you. 








The 911 dispatcher is trying to persuade Grant to put the gun down and uses simple 
reasons for the justification (“Because I don’t want you to hurt yourself’ and “Because I 
don’t want anything to happen to you”). The dispatcher did not offer any kind of 


concession or promise, but simply made a request and provided a reason. 


In the above example, the dispatcher is using reason to persuade an action 
by Grant. Reason can also be used to deter an action, just like a mother telling her child 
not to touch a hot stove. Here is another example, from a transcript provided by Paul 
Taylor (personal communication, September 20, 2009). In this example, the hostage 


taker is contemplating killing his accomplice before killing himself. 
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Hostage Taker: YEAH, WELL IF I DO ANYTHING TO <NAME+> IT’S 
BECAUSE SHE WANTS IT 


Negotiator: WELL, THAT’S DUMB. THAT’S REALLY DUMB 
Hostage Taker: DO IT TO MYSELF 


Negotiator: YOU KNOW, YOU KNOW, YOU CAN END UP IN HELL 
FOR DOING SOMETHING LIKE THAT 


Hostage Taker: TM GONNA END UP THERE ANYHOW 

Negotiator: NO, YOU NEVER KNOW 
In this example, the negotiator is not offering anything, but is merely giving a reason not 
to perform an action. Reason can play a powerful role in promoting and deterring actions 


in hostage negotiations. 
b. Reciprocity 


The principle of reciprocity is a human obligation to repay favors, gifts, 
invitations and assistance. Cialdini gives an example that shows how powerful the 
principle reciprocity can be. In 1985, Ethiopia was one of the most destitute countries on 
the planet, ravaged by drought, starvation and economic ruin. Yet in 1985, the 
government of Ethiopia sent $5,000 to the government of Mexico for earthquake 
assistance. The reason for this assistance was because Mexico had sent aid to Ethiopia 
during the Italian invasion of 1935. In Cialdini’s words, Ethiopia’s obligation to 
reciprocate for the 1935 aid “transcended great cultural differences, long distances, acute 
famine, many years, and immediate self interest.” (Cialdini, Influence: Science and 


Practice, 2001) 


Reciprocity is seemingly very powerful. People can feel obligated to 
return a favor, even if no real favor has been done. One common example is 
organizations that send people free mailing labels along with a donation card. These 
people did not ask for mailing labels (i.e., no favor was done), but many feel obligated to 
reciprocate the gesture by giving some amount of money. In some cases, the favors and 


obligation do not have to be directly connected. 
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Another example Cialdini gives is a “rejection then retreat” approach. 
Cialdini and his students masqueraded as representatives of a youth organization and 
asked random people if they would chaperone a group of juvenile delinquents to the local 
zoo. Initially, 83 percent of people asked to chaperone refused. Cialdini and his students 
then took a different approach. They asked a group of different random people for a 
larger favor first. The first request asked people to be counselors at the youth 
organization for a period of two years. As expected, the vast majority refused. After the 
initial request and rejection, Cialdini and his students asked for the people to be 
chaperones to the zoo. This time, they had a 50 percent acceptance rate (compared with 
17 percent earlier), nearly a 300 percent increase (Cialdini, Vincent, Lewis, Catalan, 
Wheeler, & Darby, 1975). By starting with a large favor, then requesting something 
smaller, Cialdini and his students had made a concession that the people in the 


experiment felt obligated to reciprocate. 


The “rejection then retreat” approach is common in police and hostage 
negotiations. This following is an excerpt from the ACLU transcript (aclusandiego.org, 
2008) between the suicidal man (Grant) and the San Diego police. The suicidal person, 


Grant, is in his apartment with a gun threatening to commit suicide. 


POLICE NEGOTIATOR: Okay. So what do you say? I think, you want to 
come outside. 


GRANT: No, no. 


POLICE NEGOTIATOR: Okay. You think you’re going to want to come 
outside later? 


GRANT: I don’t know. 


POLICE NEGOTIATOR: Okay. Can I get you at least to put the gun 
down? 


GRANT: No. 


POLICE NEGOTIATOR: How about if I get you to take the magazine or 
the clip out of the gun? 
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Note how the police negotiator makes the initial request for Grant to come outside. 
When Grant refuses, the police negotiator retreats from the original request and asks for 
something smaller by comparison (“come outside later’). After each rejection, the 


requests continue to decrease in scope. 


c. Commitment and Consistency 


Cialdini notes that “a high degree of consistency is normally associated 
with personal and intellectual strength. It is the heart of logic, rationality, stability, and 
honesty” (Cialdini, Influence: Science & Practice, 2001). It is because consistency is so 
powerful to the human condition that commitment becomes the key to the entire 
principle. If someone commits to an agreement, that person has a subconscious and 
overpowering urge to be consistent with respect to that agreement. Commitments are 


more powerful when they are public and not viewed as forced upon the party. 


Commitment and consistency play key roles in different military conflicts. 
The commitment and consistency principle was used by the Chinese with captured 
American Prisoners of War (POWs) during the Korean War. The Chinese would often 
get American POWs to admit mildly anti-American sentiments, such as “America is not 
perfect.” Once the POWs submitted, the Chinese would press them towards more 
substantial statements, including essay writing including a signature. Once an American 
POW had written the essay and signed his name, he often felt compelled to remain 
consistent with these viewpoints and change his own image to reflect what he had put on 


paper (Schein, 1956). 


Far from the Korean War, both in distance and in time, the American war 
in Iraq from 2003 to the present day has drawn many Islamic fighters from different 
nations. Some of these fighters volunteer for martyrdom operations and sign contracts. 
The following picture an example of a martyrdom contract provided by the Combating 


Terrorism Center at West Point (www.ctc.usma.edu). 
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Figure 1. | Suicide Bomber Contract 
(From the Combating Terrorism Center at West Point, n.d.) 


The translation of the document is as follows: 

In the Name of God the Most Compassionate and Merciful 

Pledge of the Martyr Brother 

1—-To be signed by the brethrens in Syria and here after their entrance. 
2-Filling the immigrant’s form. 

3—Acquiring a personal picture and a copy of the passport. 


4—Attending a training session or reading the prepared notebook on the matter. 
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5-Signing the attached pledge 
Pledge and an Agreement 


I Khalaf Ahmad Nawfal al-Rashdan, alias Abu Ahmad [TC: handwritten] 
pledge my commitment to the following: 


1-That my request to enter Iraq to conduct a martyrdom operation 
emanates from my own desire; and I have no hidden desire in becoming 
a fighter, after my entrance, and I pledge my oath on the matter. 


2-To obey, through invigorating and adverse times, my leaders and those 
who are in charge of me. 


3-In case I do retreat from executing [TC: the mission], the organization is 
not committed to me with anything except taking me out of Iraq in the 
way it sees fit. 


4-] will not reveal any information to any party, under any circumstances. 


5-I pledge my oath in the name of God the Almighty to abide by what has 


been agreed upon in this contract, and not to back out under any 
circumstance and here to, I sign. 


Name: Khalaf Ahmad Nawfal al-Rashdan [TC: handwritten] 


Number: 





Alias: Abu Ahmad [TC: handwritten] 
Signature: [TC: handwritten] 


Finger Print: [TC: a print] 


Martyrdom operations are not new to military conflicts, but note the 
signature and the thumbprint on the document. They represent a commitment by the 
individual, just like essays written during the Korean War by American POWs. The 
signature and thumbprint represent a commitment to the martyrdom operation, that is, the 
commitment not to be a fighter but a suicide operative and “not to back out under any 
circumstance.” During the course of this research, no statistics were found that describe 
the completion-withdrawal rate of suicide operatives that commit with a thumbprint and 
signature, but it is worth noting that their recruiters are using the principle of commitment 


and consistency when convincing people to perform extreme actions. 
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In police and hostage negotiations, commitment and consistency often 
arise when a deal or agreement is being stuck between two parties. In the Waco standoff 
between the Branch Davidians and the FBI (Waco Siege), there are many cases where the 
FBI agreed to perform certain tasks in exchange for the release of women and children. 
Here is an excerpt from one of the transcripts (Texas State University), where “DAVID” 


is David Koresh of the Branch Davidians and “HENRY” is an FBI negotiator: 


DAVID: I’m glad you have a concern for the children like I do because I 
have -- 


HENRY: Yes. Okay. What we will do is I will start working on this, okay. 
And it’s either going to be a video and the audio, or at least we're 
going to get radio. But I'll get you something. Or I'll go to work 
on getting you something. Okay? 


DAVID: Yeah. Let’s, let’s -- like I say you'll, then you'll know, 
understand where I'm coming from. 


HENRY: Okay. But I want you to understand, okay, I’m going to hold you 
this. You're telling me that -- 


DAVID: I -- 


HENRY: -- that you’re going to send everybody out. ’m going to want 
everybody out. 


DAVID: Okay. 
HENRY: Okay. 
DAVID: Do what you can though. 


HENRY: Okay. But send me a couple of kids. I’m going to send a car to 
pick up the two kids. 


In the above excerpt, Henry tried to get a commitment from David for the release of 
people from the compound in exchange for the FBI coordinating the release of a message 
from David onto the radio. By getting David to commit, Henry hoped that the desire to 


remain consistent would play a role in the release of the children. 
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d. Social Proof 


Social proof is another principle of persuasion based on people “viewing a 
behavior is correct in a given situation to the degree that [they] see others performing it” 
(Cialdini, Influence: Science and Practice, 2001). Some examples are rather common, 
such as advertisers using the popularity of a product to sell it to new people. A recent 
commercial by the Ford Motor Company (2010) touts the F-150 truck as the best-selling 
truck in American for 33 years. This screenshot is from the commercial posted on 
YouTube (YouTube - F-150 “33 years and counting”). The goal of this advertisement is 
to let consumers know that most Americans truck buyers buy the Ford F-150, so “you” 


(the potential consumer) should, too. 














Figure 2. F-150 Commercial (From the Ford Motor Company, 2010) 


Social proof can have some extreme effects. The Heaven’s Gate 
(Heaven’s Gate (religious group)) mass suicides grimly illustrate the power of social 
pressure to follow the crowd’s actions. A person’s actions can be heavily influenced by 
the perception of what is acceptable behavior. Cialdini gives another example of panic at 


a bank in Singapore. After the panic, it was determined that a large crowd waiting at a 
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bus stop (near the front entrance of the bank) was perceived by bank customers to be a 
line of people waiting to withdraw their money. Bank customers panicked and got in line 
to withdraw their money, mimicking the perceived actions of the crowd. This action, in 
turn, led to an even larger crowd. Humans are able to trust the collective knowledge of a 
crowd, especially if uncertainty plays a factor (Cialdini, Influence: Science and Practice, 


2001). 


In police and hostage negotiations, social proof is not always apparent, 
mainly because the situations deal with small numbers of people. There is usually only 
one negotiator and one hostage taker. However, because the situation is usually a first 
experience for the hostage taker, the negotiator can use that element of uncertainty along 
with social pressure from the hostage taker’s friends and family. The following is an 
example of social proof, where PN60 is the police negotiator and HTO1 is the hostage 
taker. The full transcript was provided by Randy Rogan (personal communication, 
October 2, 2009). 


PN60: Well things could get a lot worse. 

HTO1: Naw, I done been through the worse part. 

PN60: Oh you have aw I can’t hardly believe that. I mean you know -- [?] 
HTO1: Yeah. 

PN60: -- you’re not dying from a disease or something like that, are you? 
HTO1: No, I’m not. 

PN60: Suppose you got a healthy body and a healthy mind, right? 

HTO1: [Laughs] I wouldn’t bet on that. [Laughs] 


PN60: Well hell that’s quite a bit just that one right there. What? Well I 
don’t know what to tell you you know, you got all -- [?] 


HTO1: Huh? 


PN60: -- Your friends talkin to ya and tryin to give you advice and these 
people who know you and like you. 
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PN60 uses the uncertainty of the situation and the pressure from the hostage taker’s 
friends to convince HTO1 to give up. Social proof is one of the more difficult things to 


notice in police and hostage negotiations, due to the subtlety in which it is used. 
e. Liking 


Liking is another principle of persuasion that focuses on similarity, 
compliments, and association. Humans will act favorably towards persons with whom 
they share similar characteristics, or at least if they perceive that they share 
characteristics. These characteristics can be opinions, politics, personalities, religious 


upbringings, or social backgrounds. 


Cialdini cites several research projects in this phenomenon. One study 
showed that antiwar demonstrators were more likely to sign a petition without reading it, 
as long as the petitioner was dressed in similar fashion (Suedfield, Bocher, & Matas, 
1971). Another project found that survey petitions mailed to persons with similar names 
as the surveyor nearly doubled the compliance response (Garner, 1999). People are more 


readily compliant with people that share some kind of connection. 


In addition to similarity, positive compliments are another influential 
method for applying the liking principle of persuasion, even if the compliments are not 
accurate or true. A study was conducted with a group of subjects who were given 
different kinds of compliments and then asked for a favor. The compliments were 
positive, negative and mixed for different subjects. The results of the study were that the 
researcher with only positive compliments was liked best, even when the subjects knew 
full well that the compliment was intentional flattery meant to gain favor (Drachman, 


deCarufel, & Inkso, 1978). 


In police and hostage negotiation, the negotiator often tries to establish a 
bond with the hostage taker. This bond can be based on similar backgrounds or 
experiences. Here is an excerpt from another transcript (P. Taylor, personal 


communication, September 20, 2009). 
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Hostage Taker: SHE DIDNT WANT IT SHE DIDNT WANT IT AND I 
WASNT DOING IT FOR ME I WAS DOING IT FOR 
HER 


Negotiator: YEAH WELL YOU KNOW <NAME> THEY GET TO 
THAT AGE YOU KNOW THEY KNOW EVERYTHING 


Hostage Taker: YEAH YEAH 


Negotiator: THEY KNOW EVERYTHING YOU CANT TELL THEM 
ANYTHING 


Hostage Taker: I KNOW IT I KNOW IT I KNOW IT WELL 


Negotiator: YOU KNOW I WENT THROUGH THAT WITH MY BOYS 
YOU KNOW BUT NOW THEYRE ALL GROWN 


Hostage Taker: THE SAME THING WITH MINE 
Negotiator: MY YOUNGEST ONE IS TWENTY SIX 


In the above extract, the negotiator used the principle of liking, invoking a similar life 
experience raising children. This tactic is designed to let the hostage taker know that the 
negotiator is familiar his struggles and was an attempt to create a bond between the two 
men. This bond may have made the hostage taker more receptive to the negotiator’s 
attempts to resolve the hostage standoff. For hostage negotiations, the principle of liking 
may be important in building rapport and trust and can help the negotiator reach a 


peaceful solution. 


e. Authority 


The authority principle states that people will respond to requests if they 
feel there is an authority figure authorizing or condoning the action. A famous study 
performed by Yale university psychologist Stanley Milgram showed just how far average 
people will respond to authority, even if the authority figure is asking for something that 
is completely unreasonable (Milgram, 1974). The study had the subjects assist with a 
researcher by putting electric shock nodes on a patient who was playing along with the 
experiment. The subjects were told that the experiment was designed to test the effects of 
punishment on learning ability, and they were told to administer shocks to the patient 


when prompted by the researcher (the authority figure) based on incorrect answers to 
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questions. The electric shocks were not real, but the patient cried out in pain and begged 
for mercy and pleaded to be released. The purpose of the research was to determine how 
much pain normal people would inflict on a patient because someone in authority told 


them to do it. Cialdini writes: 


The answer is most unsettling...the [subject] was willing to deliver as 
much pain as was available to give. Rather than yield to the pleas of the 
victim, about two-thirds of the subjects in Milgram’s experiment pulled 
every one of the 30 shock switches in front of them and continued to 
engage the last switch (450 volts) until the researcher ended the 
experiment. More alarming still, almost none of the 40 subjects quit his 
job...when the victim first began to demand his release, nor later when he 
began to beg for it, nor even later when his reaction had become, in 
Milgram’s words ‘definitely an agonized scream.’ (Cialdini, Influence: 
Science and Practice, 2001) 


The explanation for these results lies in human beings’ sense of obligation to persons of 


authority. Even subjects who wished to stop the experiment continued to participate and 


obeyed the researcher until it was over. 


In police and hostage negotiations, the negotiators do not often invoke 
their authority in the law or give orders to a hostage taker. Orders might be normal for a 
typical police arrest where a police officer will give commands like “put your hands up” 
or “get down on the ground.” In the negotiations transcripts studied in this thesis, 
negotiators rarely gave orders, because the concept of authority is not clear to the hostage 
taker. The hostage takers were inclined to think of themselves as the persons with 
authority and would issue demands themselves. Here is another excerpt from the San 


Diego police standoff (aclusandiego.org, 2008). 


POLICE NEGOTIATOR: You shut up and listen to me Grant. Just shut up 
and listen to me Grant. Grant, shut up and 
listen to me Grant, Grant, Grant, Grant, people 
will not be allowed into your house. Okay are 
you going to be a coward? 


GRANT: Because of (?) 


POLICE NEGOTIATOR: Grant shut up and listen to me. Grant shut up 
and listen to me. Will you do that, keep your 
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mouth shut. You need to come out of the house 
so other people can go home. 





GRANT: You can send them home, I am not coming out. 


This police standoff did not end well as Grant ultimately committed suicide, and the 
ACLU has called for better training of the San Diego police officers for handling 
situations like this one (ACLU Calls for Training of Police Negotiators After Suicide, 
2008). This example is not meant to deride the police negotiator’s actions, but to 
illustrate how the concept of authority can be used in negotiations, both successfully and 


unsuccessfully. 


Instead of giving orders to a hostage taker, negotiators often will take an 
indirect approach to the authority principle. One such approach is to use family members 
of the hostage taker to persuade compliance with the police. The authority figures do not 
have to be a direct part of the situation; they can simply be referenced. Here is another 


section of the San Diego police transcript (aclusandiego.org, 2008). 


POLICE NEGOTIATOR: So killing yourself is going to do some good? 
GRANT: Well, obviously, I don't have to think about anything else. 


POLICE NEGOTIATOR: You don't, but that's the easy way out. What 


about your mother _and father, your brother, 
your friends. Don't let them down. What’s that? 


GRANT: What about them? 
POLICE NEGOTIATOR: Well, how upset are they going to be? 
GRANT: I don't want to upset my friends. 


POLICE NEGOTIATOR: And you're going to let all those people down 
like that? 


The negotiator was appealing to Grant’s sense of duty and obligation to his parents so he 
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would not “let them down.” The authority principle, while indirect, can be exercised by 


negotiators to convince hostage takers to comply with requests. 
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f Scarcity 


The principle of scarcity states that people are more likely to be influenced 
if they feel circumstances are fleeting or resources are limited. There is a direct 
correlation between the scarcity of an item and its perceived value. When an item 
becomes scarce, it becomes less accessible to people, meaning people lose the freedom to 
acquire it. People respond to this loss by wanting those freedoms even more (Cialdini, 
Influence: Science and Practice, 2001). 


Most people see this principle everyday in some form of advertising. 
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Things like “limited time offer,” “sale ends Friday,” and “call now while supplies last” 
are all examples of attempted persuasion with scarcity. Basic free markets operate on 
supply and demand, with supply representing the scarcity of a commodity. Commodities 
markets can have huge influences on peoples’ behaviors. When Hurricane Katrina 
landed on the gulf coast in August of 2005, it caused significant damage to the American 
gasoline infrastructure. Figure 3 shows that gasoline prices jumped almost 18 percent in 


one week (U.S. Retail Historical Gasoline Prices). 





Weekly U.S. Regular Conventional Retail Gasoline 
Prices (Dollars per Gallon) 
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Figure 3. U.S. Gasoline Prices from August 2005—October 2005 


Some gas stations ran out of fuel to sell to customers. The shortage of 


gasoline was attributed to both a reduction in supply as well as a customer swarm on the 
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remaining amounts at gas stations (Goo & Blum, 2005), fearing that there would be no 
gasoline available in the area. People consciously and unconsciously react to scarcity and 
“compliance practitioners’ reliance on scarcity is frequent, wide-ranging, systematic and 


diverse” (Cialdini, Influence: Science and Practice, 2001). 


In the police and hostage negotiations, scarcity is a method of persuasion 
used by both law enforcement and the hostage takers. Many demands from hostage 
takers come with deadlines and the promise of horrific consequences if their demands are 
not met. Here is an example from another transcript (R. Rogan, personal comunication, 
October 2, 2009). Note the personal information such as personal names have been 
removed for distribution reasons. 

PN6O: .. . to <HTO1 First Name>. (pause) 

HTOl1: Yes, sir. 

PN60: Get the cigarettes all right? 

HTO1: Yes, sir. 

PN60: Everybody's feeling all right out there? 

HTO1: .. . to be. 

PN60: Your lawyer's enroute. Should be here in about 15 minutes. 


HTO1: I'm setting a 5:00 deadline. 





PN60: Uh, <HTO1 First Name>, we can't, uh, be sure that we're gonna 
have all of the answers by 5:00. Uh, let's not set a deadline on it. L- 
let's, uh, play it by ear and see how things go. We've play, tried to 
play it pretty straight with you, haven't . . . and, and it's not going 
to resolve the matter. We've got to find out primarily before we can 
get answers what your intentions are as far as what if <Prisoner 
Last Name> fails, goes with you, what if he refuses to go with you. 


HTO1: .. . not the point either. The point is I've set some demands, and if 
they're not met, then the people are going to die. 





This example is very common in hostage negotiations. The hostage taker sets a deadline 
that will expire and expects the negotiator to acquiesce to the demands. While demands 
are not always met, many negotiators are persuaded into some form of action to keep the 


hostage taker from hurting anyone. 
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The principle of scarcity is more subtlety applied by the negotiator. The 
negotiator is wise not to set deadlines with the hostage taker, because the situation can 
deteriorate rapidly. Often the approach is more subtle, focusing on circumstances instead 
of time. One tactic used is letting the hostage taker know that the situation is not as bad 
as it seems, but it could get worse. Here is another example (P. Taylor, personal 


communication, September 20, 2009): 


Hostage Taker: YOU SAYING IF 1M CHARGED FEDERALLY? 


Negotiator: OK, YOU ARE CHARGED FEDERALLY IF THAT’S 
WHAT YOU WANT. I MEAN YOU KNOW YOU’RE 
THE ONE THAT’S CALLING THE SHOT, BUT YOU 
HURT SOMEBODY OVER THERE, WELL, YOU’RE 
PROBABLY GONNA GO TO <PLACE>. THAT’S FOR 
<EXPLETIVE> SURE 


Hostage Taker: I AINT GONNA HURT NOBODY 


Negotiator: BECAUSE IF YOU GIVE UP RIGHT NOW, YOULL 
PROBABLY GET CHARGED FEDERALLY. THAT’LL 
BE THE DEAL. BUT IF YOU DRAG THIS THING ON 
AND ON AND ON AND HURT THAT WOMAN OR 
HURT ANYBODY ELSE... 





In this example, the hostage taker would rather be charged federally than by the state. 
The negotiator realizes this desire and says that outcome is a possibility, but it is a 


fleeting opportunity. 
g. Simultaneous Forms of Persuasion 


Seven principles of persuasion have been discussed: reason, reciprocity, 
commitment and consistency, social proof, liking, authority, and scarcity. The transcript 
excerpts provided were selected because they were clear-cut examples of each form. 
These categories of persuasion are not mutually exclusive. It is possible for a persuasion 
attempt to fall into multiple categories. This example comes from another transcript. (R. 


Rogan, personal communication, October 2, 2009) 
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Hostage Taker: How much time you talking about? 


Negotiator: We don't know, but, uh, let's get the food order squared away. 


How ‘bout if you let the stewardess off, uh, to get the food, 
and, uh, she, and then we'll let somebody bring the food up to 
the plane? 


The underlined section could be attributed to commitment and consistency or reciprocity. 
One view is that the negotiator is trying to get the hostage taker to commit to an exchange 
of food for stewardesses. Another view is that the negotiator is providing food and wants 
the hostage taker to reciprocate by releasing the stewardesses. Either one or perhaps both 
are correct. This thesis tags the individual persuasion principles of utterances in 
conversation, but these principles will be combined in each utterance to determine if it is 
meant to be persuasive. In the above example, if one annotator marks the utterance as 
“commitment/consistency” and another annotator marks it is as “reciprocity,” the 
differences will be noted, but there will be agreement that the statement is meant to be 
persuasive. Moreover, annotators are allowed to mark utterances with multiple 


persuasion tags if they think it falls into multiple categories. 
D. ANNOTATION EVALUATION METHODS 


In evaluating annotated transcripts, the Cohen Kappa (Cohen, 1960) score is a 
useful tool that can take into account the agreement on both persuasive and non- 
persuasive utterances while discounting the probability of random agreement. The kappa 
score is defined as 


k= ObservedA greement — ChanceAgreement _ Pr(a)—Pr(e) 


NonChanceA greement 1—Pr(e) 


where Pr(a) is the relative observed agreement and Pr(e) is the hypothetical probability of 


random agreement. Table 2 shows the hypothetical results of an annotated transcript. 
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Table 2. | Agreement Matrix for Two Annotators 


Annotator B 


























Persuasive Non-Persuasive 
Persuasive PaPp PaNp 
Annotator A 
Non-Persuasive NaPp NaNp 
PaPp= the total number of utterances that both annotator A and B marked persuasive. NaPp=the 


total number of utterances that annotator A marked non-persuasive but annotator B marked persuasive. 


PNgz = the total number of utterances that annotator A marked persuasive but annotator B marked 
non-persuasive. 


NaNsg = the total number of utterances that both annotator A and B marked non-persuasive. 


Assuming the annotators tagged the data independently, the probability of 
agreement, Pr(a) is defined as the percentage of the total utterances where both annotators 
agree: 

PaPs + NaNe 

Pr(a) = —————_ 
where n is the total number of utterances in the transcript. To calculate chance 
agreement, each annotator must be scored on the number of utterances they marked 


persuasive and non-persuasive with respect to the transcript: 


Pr( Pa) = PaPs+ PaNe Pr( Px) = PaPs+ NaPp 
n 

Pr(Na) = NaPs + NaNe Pr(Ns) = PaNs + NaNe 
n 


Thus, the chance of random agreement (assuming independence) is defined as: 


Pr(e) =[Pr(Pa) x Pr(Ps)]+[Pr(Na) x Pr(Ns)] 


With Pr(a) and Pr(e) defined, Cohen’s Kappa score can now be calculated with the 


formula previously mentioned: 
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ObservedA greement — ChanceAgreement _ Pr(a)—Pr(e) 
NonChanceA greement 1—Pr(e) 


Figure 4 shows an interpretation of the kappa score (Landis & Koch, 1977). 








Kappa Statistic Strength of Agreement 
<0.00 Poor 
0.00-0.20 Slight 
0.21-0.40 lair 
0.41-0.60 Moderate 
0.61—0.80 Substantial 
0.81-1.00 Almost Perfect 











Figure 4. _ Interpretation of Cohen's Kappa Measurement (From Landis & Koch, 
1997, p. 165) 


This research will use the kappa score and the measurements in the previous 


figure to evaluate the annotation scheme. 
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HiIl. CODEBOOK AND EXPERIMENT DESIGN 


A. INTRODUCTION 


In this chapter, we present our annotation scheme for persuasion and the coding 
guidelines developed. The annotation scheme uses the seven categories discussed in 
Chapter II: reason, reciprocity, commitment and consistency, authority, liking, scarcity 
and social proof. In addition, commitment and consistency were split into separate 
categories for a total of eight possible tag markers for each utterance. We use the eight 
categories to tag the data using different annotators, and then compare the results using 


Cohen’s Kappa score. 
B. INITIAL MODEL AND FIRST ANNOTATION OF DATA 


A group of three annotators attempted to tag 37 transcripts with the eight 
categories of persuasion discussed in Chapter II (reason, reciprocity, commitment, 
consistency, liking, scarcity, authority, and social proof). Annotator A tagged 16 
transcripts, Annotator B tagged 17 transcripts and Annotator C tagged 16 transcripts. 
There was an overlap of 12 transcripts in total with each pair of annotators sharing four 
transcripts. These 12 overlapping transcripts were used in the analysis portion of this 


thesis. 


The annotators tagged each utterance in their transcript. An “utterance” is simply 
defined as a stretch of speech containing at least one word. For this work, we treat each 
“turn” in the transcription as an utterance. The following example from the Waco 


transcripts (Texas State University, 1993) contains six utterances: 


Utterance 1: JIM: Is that okay? 
Utterance 2: DAVID: That's fine. 


Utterance 3: JIM: Okay. Now tell everybody we'll get, we got -- what 
we're going to work on the next few minutes is moving the 
armored ambulance and the buses. Okay? 
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Utterance 4: DAVID: Okay. You're going -- 
Utterance 5: JIM: So when they -- 
Utterance 6: DAVID: -- to bring those up then now, right? 
There are three utterances from “Jim” and three from “David.” Utterances do not have to 


be complete sentences, and words like “uh-huh” and “okay” are counted as utterances 


like in the following example (Texas State University, 1993) from the Waco transcripts: 


Utterance 1: JIM: You know. Just so we have no mistakes. If you see the 
armored ambulance pull up, you can call me. 


Utterance 2: DAVID: Okie doke. 
Utterance 3: JIM: Hey, your message got out great. 


In this example there are three utterances with two by “Jim” and one by “David.” 


David’s answer to Jim is simply “okie doke,” but it counts as an utterance. 


1. Initial Codebook 


a. Reason 


Reason was tagged when someone in the conversation gave a justification 
for some action or request. The following examples show the same request but only one 


has a reason: 


Utterance 1: I want you to put the gun down 


Utterance 2: I want you to put the gun down because I don’t want to see 
you get hurt. 


Only Utterance 2 would be tagged with reason by the annotators, even though both 
utterances are requesting the same thing. Annotators were told to tag an utterance with 


reason only when a request was made and there was a justification given. 
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b. Reciprocity 


The reciprocity principle has many forms so the annotators were told to 


use the following guidelines: 


ke 


Any instance where one speaker implies the other is somehow indebted to 
him or her. This indebtedness can come from a previous solicited or 


unsolicited favor. 


Any series of requests that starts with something large, then gets smaller 
by comparison. In cases like these, annotators were instructed not to tag 
the original request with reciprocity but all subsequent smaller requests 
instead. Here is an example, seen previously in Chapter II 


(aclusandiego.org, 2008), of such an exchange: 


Utterance 1: POLICE NEGOTIATOR -- Okay. So what do you say? I 


think, you want to come 
outside. 


Utterance 2: GRANT -- No, no. 


Utterance 3: POLICE NEGOTIATOR -- Okay. You think you’re going to 


want to come outside later. 


Utterance 4: GRANT -- I don't know. 


Utterance 5: POLICE NEGOTIATOR -- Okay. Can I get you at least to 


put the gun down? 


Utterance 6: GRANT -- No. 


Utterance 7: POLICE NEGOTIATOR -- How about if I get you to take 


the magazine or the clip out of 
the gun. 


In this example, Utterance 1 would not be tagged with reciprocity, but Utterance 3, 


Utterance 5, and Utterance 7 would be tagged. 


Cc. Commitment 


For the principle of commitment, annotators were told to tag any instance 


where one party tries to make a deal with the other party. In instances of commitment, 
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there is usually an if-then exchange. In hostage negotiations, common examples are 
exchanging food or favors for hostages. Annotators were asked to tag any instance of an 
agreement and also if there was prodding by one party to recognize the agreement. In the 


following exchange, both Utterance 2 and 4 would be tagged with commitment. 


Utterance 1: Pl -- Okay, I'll give you the pack of smokes, but you need to 
let the lady go, okay? 


Utterance 2: P2 -- Okay, just send someone up here and throw the pack 
through the door, and I'll send her out. 


Utterance 3: P1 -- Okay, so we have a deal, right? 


Utterance 4: P2 -- yeah, okay. 
d. Consistency 


Consistency and commitment are similar but annotators were asked to 
distinguish the two in the tag system. Consistency can only be applied once a 
commitment has been made. Any reference to a previous commitment to persuade one 
party to hold up their part of an agreement should be marked with consistency. For 


example: 


Utterance 1: Negotiator -- Okay, you’ve got the smokes, now let the lady 
go. 


Utterance 2: Hostage Taker -- I don’t know, she’s my only leverage. 


Utterance 3: Negotiator -- Hey man, you gave me your word that you 
would let her go. C’mon, do the right thing. 


In this example, Utterance 3 is marked with consistency because the negotiator is 


reminding the hostage taker of a previous commitment. 
e. Scarcity 


Annotators were told to tag any utterance with scarcity if the utterance 


made reference to a fleeting situation. In other words, if negotiators talked about the 
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situation having the potential to get worse and that the hostage taker should surrender, 


then that utterance should be tagged with scarcity. 
f. Liking 


When tagging the principle of liking, annotators were told to watch for 
cases where the negotiator tries to note similar life experiences with the hostage taker. 


Things like similar jobs or familiarity with family issues were examples. 
g. Authority 


Annotators were told to tag any instances of authority figures with the 
hostage takers. If the negotiator was able to bring a parent into the conversation to 
convince the hostage taker to end the situation peacefully, annotators were told to mark 


the requests by the parents with the authority tag. 
Ze First Annotation 


A persuasion tag vector was created and assigned to each utterance by the 
annotators. The annotators were asked to mark each utterance with any of the agreed 
upon principles. If there were multiple kinds of persuasion in an utterance, annotators 
were asked to rank them in order of importance. If an utterance contained at least one 
persuasion tag, then it was labeled persuasive. Figure 5 is an example of the tagging 
scheme from a bank robbery transcript (R. Rogan, personal communication, October 2, 


2009). 
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PERSUASION TAGS 


Utterance REASON | RECIPROCITY | COMMITMENT| CONSITENCY | LIKING scarcmy | aurHoriry | social proor| PERSUASIVE? 
Reporter's Last Name> on the phone. 0 
00094 PN1: OK. Now let me ask youa 

question. Now, if Mr. <News Reporter's 

Name> asked you to let these kids out, 

and asked you how these kids are, will 

you answer his questions? 1 1 
00096 PN1: Ah- | don't think-ah- my boss, 

ok, my higher up, let him come in there. 

| don't- just for his safety also. But we 

have ta- we have ta- we'll have to work 

that problem when we come to it, OK? z 







































Figure 5. Sample of a Transcript Annotation 


Utterances 94 and 96 are marked with commitment and authority, respectively, so 
they each have a “1” in the persuasive category on the right. The other utterances are not 


persuasive, so they have a “0” in the persuasive category. 


3. Evaluating the Initial Annotations 


a. Initial Results 


When two annotators tagged the same transcript, agreement was measured 
using the Cohen Kappa score described in Chapter II. We also considered Krippendorf’s 
alpha measurement, but given the annotators’ task, Cohen’s Kappa is more appropriate, 
and the alpha measurements are nearly the same as the kappa scores (Krippendorf, 2007). 
Cohen’s Kappa measure is useful because it takes into account agreement on both 
persuasive and non-persuasive utterances. The initial results were interesting but only 
showed moderate agreement, which is described in detail in Chapter IV. The initial 
results led annotators to consider whether or not the discrepancies in agreement were a 
training or model issue. In response to these questions, annotators took a 600 utterance 


transcript and tagged the first 300 utterances together, justifying with each other their 
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reasons for tagging certain utterances as persuasive. The goal was to tag the first 300 


utterances together for training, and then tag the last 300 individually to compare results. 
b. Category Problems with the Initial Model 


One of the immediate observations between the annotators was the large 
discrepancy on deciding what the category “reason” encompassed. One of the reasons 
for the differences was the subjectivity of the criteria defining the principle of reason. 
Based on this difficulty, the category was eliminated, but this adjustment still left 


annotators with a few issues. 


One issue was that annotators could agree that certain utterances were 
meant to be persuasive, but they sometimes felt the persuasion attempt did not fit into any 
of the Cialdini categories. The goal of this thesis was to detect persuasion in 
conversation, and if two annotators agree that an utterance is persuasive but cannot find a 
category that fits the persuasion description, then there is a problem. Rather than ignore 
these persuasion attempts, a new category was created, generically called “other.” The 
“other” category was used by annotators to tag any persuasive utterance that did not fit 
into the Cialdini categories. The following exchange is a good example in the use of the 
“other” category. In this example, Steve is a Branch Davidian involved in the standoff at 


Waco and Jim is a negotiator (Texas State University, 1993). 


Utterance 1: STEVE - The news media was told to get out of the way and 
to back way off, much farther back than what they 
are even now. Why is that? 


Utterance 2: JIM - Nobody's coming in there. A thousand percent absolute 
guarantee. I'm promising you, it's true, okay? Do we got 
those kids? Okay. Steve? 
Annotators agreed that utterance 2 by Jim was meant to be persuasive with 
the use of words like “guarantee” and “promising”, but the utterance did not fall under 


any of original categories. In the new scheme, this utterance would be tagged with 


“other.” 
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Another problem that annotators discovered while tagging together was 
that some of the persuasion principle definitions were imprecise. This lack of precision 
can be correlated to the fact that there are over 25,000 utterances, and even with a pilot 
study, not all instances were discussed before the full annotation. During training, the 
key aspects of each principle were explained, but only one or two examples were given. 
Once annotators were able to view many instances of persuasion, they tended to slightly 
different definitions of the persuasion categories. However, while tagging the 300 
utterances together, the annotators were able to arrive at more precise definitions on 


which they could all agree. 


An additional problem was that some annotators were only tagging 
persuasion when the negotiator was speaking and did not consider that persuasion 
can occur on both sides of the negotiation. After discussion between the annotators, 
an agreement was reached to tag any instance of persuasion, regardless of the person 
speaking. The hostage takers can be persuasive in their own way by setting 
deadlines and demanding concessions in exchange for commitments, and the new 


tagging system would have to reflect that fact. 
C REVISED MODEL AND SECOND ANNOTATION OF THE DATA 


Based upon their discussions, the annotators agreed that certain instances of 
conversation should be tagged with persuasion, and new, more precise definitions were 


created. 
1. Reciprocity 


1. Look for instances where one party tries to make the other feel indebted to 
them 
This is not a regular commitment (“If you do this, P’Il do that’) 
b. The favor done should not be part of a previously established 
agreement, otherwise reminding the person of the agreement 


would fall under consistency 
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C In negotiations, common examples include instances where the 
negotiator relays to the hostage taker how hard he is working for 


him but needs something in return 


2. Look for cases where the negotiator asks favor, which is then rejected, but 
then follows with a smaller favor. In cases like this one, do not tag the 


original request with reciprocity, but only the subsequent smaller requests. 
2. Commitment 


1. Look for any kind of deal making (“If you do this, I'll do that...) 


2: May have to be taken in context as in the following excerpt (Texas State 


University, 1993): 


Utterance 1: JIM: As soon as we get these kids hooked up, I'm going to go 
back and talk to these commanders -- 


Utterance 2: STEVE: All right. Okay. 

Utterance 3: JIM: --about that perimeter motion, okay? 

Utterance 4: STEVE: Right. 

Utterance 5: JIM: And, and some of these issues we've discussed. 
Utterance 6: STEVE: Right, okay. 


Utterance 7: JIM: I've got to round them up. There's a bunch of them in 
here. 


Utterance 8: STEVE: Okay. 
Utterance 9: JIM: And -- 
Utterance 10: STEVE: And call me back then? 
In this case, annotators agreed that Steve wants a commitment from Jim to call him back 


after he speaks with his commanders. Utterance 10 would be tagged with commitment, 


but there is no specific “If-then” phrase. 


Here is another example (Texas State University, 1993) of understanding 


commitment within the context of the conversation: 


37 


Utterance 1: HENRY - Okay. So, you know, I would like for you, you 
know, in order for me to, to, to be able to, to sell 
all of this to my bosses, I need a little bit of help 
from you. Now, what I can do, or what I'd like to 
do is let me talk to them in terms of selling this 
to KRLD and maybe we can have KRLD then go 
national during prime time. And if we do that, 
will you agree to come out at that time? 


Utterance 2: MR. KORESH - When you've done the national. 
Utterance 3: HENRY — Okay. 


Utterance 4: MR. KORESH - You can do KRLD and I'll still send the 
children. 


Utterance 5: HENRY - Okay. 
Utterance 6: MR. KORESH - You know. 
Utterance 7: HENRY - All of them? 


In this example, Utterance 1 is tagged with commitment (“if we have the tape 
played nationally during primetime, will you agree to come out?’) because it fits the 
commitment guidelines. In addition, Utterance 7 is tagged with commitment because it 
represents a different commitment (“If we play the tape on KRLD (the local station) will 
you send out all of the children?’”’). Utterance 7 is an example of a commitment without a 
literal “‘if-then” structure, and the commitment must be recognized by the context of the 
conversation. This type of tagging is one of the more difficult tasks for annotators, 
because it requires them to understand the context of the utterance. Utterance 7 (“All of 
them?’’) by itself is just a question, not a commitment attempt, but taken in context of the 
negotiator trying to ensure the release of the children, it becomes a commitment attempt. 
It is worth noting that, in theory, one could view utterance 7 as an information request, 
not a persuasion attempt. If this was the first time David had ever mentioned the deal 
(children released for tape played locally) to Henry, Henry may have been just trying to 
establish what exactly the parameters of the exchange were. But in this instance, and in 
the context of the conversation, multiple annotators saw utterance 7 as a persuasion 


attempt. 
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>. Commitment can also be one party emphasizing an agreement has been 
made. Getting someone to recognize a commitment has been made is 
different from reminding them of a previous agreement, which is covered 


in the consistency tag. 
3; Consistency 


1. When one party (party A) makes a reference to a previous commitment by 
Party B in order to persuade Party B into some action or request. Here is 


an example (Texas State University, 1993). 


Utterance 1: HENRY - Okay. So, you know, I would like for you, you 
know, in order for me to, to, to be able to, to sell 
all of this to my bosses, I need a little bit of help 
from you. Now, what I can do, or what I'd like 
to do is let me talk to them in terms of selling 
this to KRLD and maybe we can have KRLD 
then go national during prime time. And if we 
do that, will you agree to come out at that time? 


Utterance 2: MR. KORESH -: When you've done the national. 
Utterance 3: HENRY - Okay. 


Utterance 4: MR. KORESH -: You can do KRLD and I'll still send the 
children. 


Utterance 5: HENRY - Okay. 
Utterance 6: MR. KORESH -: You know. 
Utterance 7: HENRY - All of them? 


Utterance 8: MR. KORESH -: Not all of them yet. Like I say, when you 
go national, then they'll all come, all of us 
come. 


This example is expounded from the previous commitment principle example. 
The negotiator (Henry) tries to get Koresh to commit to letting all of the people go 


(Utterances 1 and 7). However, Koresh himself uses the principle of consistency in 
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Utterance 8 to persuade Henry to stick to the original plan. Koresh reminds Henry of a 
previous agreement (a national radio address in exchange for full surrender) in order to 


make Henry abide by the original plan. 


1. When one party (Party A) makes reference to a want or need by Party B in 
order to influence some kind of belief or action. In the following example 
(R. Rogan, personal communication, October 2, 2009), HTO1 has taken 
control of an airplane and is demanding they bring a prisoner to the 
airplane so he can talk to him. ON82 is the attorney for HTO1 and he is 
asking what HTO1 needs: 


Utterance 1: HTO1 - Need an honorable way out. 
Utterance 2: ON82 - Say it again. I didn't read you. 
Utterance 3: HTO1 - [need an honorable way out. 


Utterance 4: ON82 - [ ] honorable way, <HTO1 First Name>. We're not 
dealing with a, we're dealing with innocent people. 
That's not the honorable way. Uh, you and I have 
got to figure an honorable way out. Tell me what 
you mean by honorable way out. 


<conversation goes on> 


Utterance 5: 0478 ON82 - Okay, <HTO1 First Name>, now one thing they 
need, and I think this is reasonable. It, you, you 
tell me if you don't think so. If I can get 
<Prisoner Last Name> out here, I need a show 
of good faith that you're, that we're dealing as 
man to man. Release the two women. Keep the 
men and the pilots but release the two women 
so I've got something to bargain with. I can't 
just tell these guys to shove it. I gotta have 
something to bargain with. 


Utterance 6: HTO1 -. . . got it. 


Utterance 7: ON82 - Okay. Let the two stewardesses go. I'll kick their 
<expletive> 'til we get <Prisoner Last Name> out 
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here. Okay? Hang in there with me. Okay? Let me 
get some instructions on how to handle it. 


Utterance 8: HTO1 - I'll let the stew-, stewardesses go as soon as 


<Prisoner Last Name> comes. 


Utterance 9: ON82 - <HTO1 First Name>, we gotta have a show of good 


faith on your part first. I, I'm raising <expletive> to 
get him out to you. They're trying to raise the 
Attorney General in Washington. Let the girls go. 
You got the men left. That's honorable. We talked 
about children awhile ago. The girl's got a four- 
year-old daughter. She's a widow. There's nobody 
else but her. Let her off, and I'll kick <expletive> 'til 
I get something done. Okay? 





In Utterance 2, HTO1 expresses his need for an honorable way out and ON82 uses 


consistency in utterance 9 to convince HTO1 to release the two stewardesses, claiming 


that it is the honorable thing to do. 


4. 


Scarcity 


Any utterance which implies explicitly or implicitly that time is a factor 
a. A hostage taker setting a deadline for demands to be met 


b. A negotiator claiming that a situation may get worse in the future 


unless the hostage taker acts now 


Any time “secret” information is used to influence another party’s 
decision. The following example (P. Taylor, personal communication, 


September 20, 2009) is from a prison riot transcript: 


Utterance 1: Hostage Taker - Now you have to your word all night long as 


far as I can see are you serious are about 
that? 


Utterance 2: Negotiator - I am <Hostage Taker Name> okay. 


Utterance 3: Hostage Taker - Everybody else 


Utterance 4: Negotiator - let you all in on a little secret okay? These 


guards out here, they’re pushing me to get 
something done and I am trying to hold them 
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back because I know you all are up to your 
word; whatever you all gonna tell me you all 
gonna do, you all gonna do it, but these guys — 
they’re, you know, they’re nervous just like 
<name> is and I’m trying to calm them down. 
I got, man, I have to move offices. You notice 
how quiet it is now? 


In Utterance 4, the negotiator uses “secret” information and the principle of 


scarcity to let the hostage taker know that the guards are getting anxious and that he 


needs to resolve the situation soon before things get worse. 


5. 


Liking 


Any kind of compliment in order to influence decisions. Most times, 
annotators agree that influential compliments are given from the 
negotiator to the hostage taker, however there can be cases where the 


hostage taker uses the principle of liking to influence the negotiator. 

a. Can be brown-nosing and insincere 

b. Words and phrases like “buddy”, “friend” 

Any reference to similar life experiences 

Any expression of affection towards one party; phrases like “I like you” 
and “it’s important to me that you make it out of this” are examples. 
Authority 

Appealing or making reference to a higher authority or expert in order to 
persuade or influence one party’s beliefs or actions 


a. A negotiator making reference to his bosses needs in order to 


influence the hostage taker 


b. A negotiator making reference to an authority figure in the hostage 


taker’s life like a parent or older sibling 
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2. Any request for action or belief to a hostage taker from an authority figure. 
If an utterance like “Please put the gun down and come outside” comes 
from the negotiator, then that utterance is not persuasive. However, if the 
same utterance were said by the hostage taker’s mother, then the utterance 
would be persuasive, because the mother is an authority figure. This type 
of tag requires the annotators to understand the context of the situation and 


the relationships between the parties. 
Ts Social Proof 


Lk. Any reference to what is normal or customary in situations (a social 
norm). The negotiator might make reference to what a judge would 


normally do in order to influence the hostage taker. 


2. Any appeal to what a group thinks the person should do. A negotiator 
might make reference to the hostage taker’s friends or family, claiming 
they all think he or she should give up. In the following example (R. 
Rogan, personal communication, October 2, 2009), the negotiator (PN60) 


is trying to convince the hostage taker (HTO1) to give up. 


Utterance 1: PN60: Suppose you got a healthy body and a healthy mind, 
right? 


Utterance 2: HTO1: [Laughs] I wouldn't bet on that. [Laughs] 


Utterance 3: PN60: Well hell that's quite a bit just that one right there. 
What? Well I don't know what to tell you know, you 
got all '- [?] 


Utterance 4: HT0O1: Huh? 


Utterance 5: PN60: -- your friends talkin’ to ya and tryin’ to give you 
advice and these people who know you and like 
you. 


In Utterance 5, the negotiator makes reference to the hostage taker’s friends and 
their opinion that he should give up. This kind of technique amounts to peer pressure and 


is a good example of the principle of social proof. 
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8. Other 


The “other” category is a catchall for any utterance that annotators view as 
persuasive but does not fit into the Cialdini model. This category is hard to define 
specifically because it is an open-ended group. However, annotators were able to agree 


on certain examples that qualify for the category. 


1. A request from a non-authoritative family member like a sibling. 
2. An appeal to the hostage taker to think about their children 
eA An emphatic plea by the negotiator using words like “guarantee,” 


“absolutely,” etc., in order to gain trust 
4, Reasons why certain actions should be performed (justifications) 


Ds Reminding of the trust built up between the two parties. For this type, 
annotators settled on the term “self-consistency” but did not want to 
include this in the consistency category, because it involved issues of trust 


rather than specific commitments. 


6. Any other utterance that an annotator thinks is persuasive but does not fit 


under the Cialdini categories. 


Annotators were told not to force persuasive utterances into the Cialdini 
categories. If they felt that something was persuasive but did not fit into the normal 


categories, then they were instructed to tag it with the “other” category. 
D. PILOT TESTING OF THE NEW MODEL 


After tagging the first 300 utterances of the training transcript together, agreeing 
on more specific category definitions, eliminating the reason category and adding the 
extra “other” category, the annotators tagged the remaining 300 utterances separately. 
When comparing the 300 utterances that were individually tagged, the results showed a 


dramatic increase in agreement. 


Figure 6 gives a visual representation of agreement between the three annotators. 
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New Model: Individually Tagged Utterances 





Persuasive 


Not 
Persuasive 


@ Annotator A 
Ute 
@ Annotator B era 
Ne Ny, 


® Annotator C 











Figure 6. | Graphical Representation of Agreement Between Annotators for 
Individually-tagged Utterances in the Test Transcript in the Re-designed Model 


This figure shows the broad agreement between the three annotators using the 
new model. Visually, there are definitive persuasive sections of the conversation that all 
three annotators were able to identify. Based on these results, the annotators reran the 


experiment using the revised model, and the new results are detailed in Chapter IV. 
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IV. RESULTS AND ANALYSIS 


A. INTRODUCTION 


This chapter discusses the results of our experiment described in Chapter III and 
is divided into three sections. The first section contains the results from the initial model 
and annotation experiment. The second section documents the results from the pilot 
study of the refined model where annotators tagged the first half of a 600-utterance 
transcript together and the last half individually. The last section contains the results of 
the new model with the altered categories and updated codebook. As discussed in 
Chapter II, the measurement for agreement is determined using Cohen’s Kappa score, 


meaning a higher kappa score implies better agreement. 
B. INITIAL MODEL RESULTS 


Each annotator was asked to label each utterance in an assigned transcript as 
persuasive or non-persuasive based on these eight categories: reason, reciprocity, 
commitment, consistency, liking, scarcity, authority, or social proof. Tables 3 through 11 
detail the results of the experiment, with each table representing one transcript. The 
numbers in the cells represent the number of utterances given their label in the table. For 
example in the following table, the number 224 means there were 224 utterances in the 
Taylor! transcript that Annotator A and Annotator B labeled as non-persuasive. The 


other numbers were calculated similarly given their corresponding labels. 


Table 3. Results for Taylor! Transcript (Initial Model) 














; Annotator B 
Transcript: 
Taylor1 Persuasive Non-Persuasive 
Persuasive 4 5 
Annotator A 
Non-P, i 
on-Persuasive g 94 




















Kappa 0.353 
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Table 4. 


Results for Waco_11A_3 Transcript (Initial Model) 


Annotator B 














Transcript: 
Waco_11A 3 Persuasive Non-Persuasive 
Persuasive 
Annotator A ; 9 1 
Non-Persuasive 0 104 








pape 0.943 
























































Table 5. = Results for Waco_12B_2 Transcript (Initial Model) 
Annotator B 
Transcript: 
Waco_12B 2 Persuasive Non-Persuasive 
Persuasive 
Annotator A ; 3 2 
Non-Persuasive ) 247 
noes 0.592 
Table 6. — Results for RoganF Transcript (Initial Model) 
Annotator B 
Transcript: 
RoganF Persuasive Non-Persuasive 
Persuasive 
Annotator A 39 56 
Non-Persuasive 6 AA5 
pee 0.380 
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Table 7. — Results for Waco_14 Transcript (Initial Model) 


Annotator B 











Transcript: 
Waco_14 Persuasive Non-Persuasive 
Persuasive 
Annotator C 5 7 





Non-Persuasive 





8 905 











Kappa 0.392 


Table 8. —_ Results for RoganA Transcript (Initial Model) 








Annotator B 

















Transcript: 
RoganA Persuasive Non-Persuasive 
Persuasive 
Annotator C ; 11 7 
Non-Persuasive 20 331 
ea 0.398 











Table 9. — Results for RoganB Transcript (Initial Model) 





Annotator C 


























Transcript: 
RoganB Persuasive Non-Persuasive 
Persuasive 
Annotator A 95 97 
Non-Persuasive 149 1781 
ae 0.372 
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Table 10. 


Results for Waco_11B_1 Transcript (Initial Model) 


Annotator C 









































Transcript: 
Waco_11B_1 Persuasive Non-Persuasive 
Persuasive 
Annotator A ; 1 0 
Non-Persuasive 0 A0 
Kappa 10 
Table 11. Results for Waco_13_1 Transcript (Initial Model) 
Annotator C 
Transcript: 
Waco_13_1 Persuasive Non-Persuasive 
Persuasive 
Annotator A ; 2 0 
Non-Persuasive 0 7 
Kappa 10 














utterances in the transcript which is detailed in the Table 12. 
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The kappa scores showed mostly fair agreement, but there were some cases of 


kappa scores above 0.9. These high kappa scores seem to correlate with a low number of 











Table 12. Transcript Kappa Scores & Agreement (Initial Model) 


























| TRANSCRIPT | UTTERANCES | = KAPPA ~~ | AGREEMENT | 
Taylor! 241 0.353 Fair 
Waco_l1lA 3 134 0.943 Almost Perfect 
Waco_12B 2 254 0.592 Moderate 
RoganF 562 0.380 Fair 
Waco_ 14 925 0.392 Fair 
RoganA 269 0.398 Fair 
RoganB 2122 0.372 Fair 
Waco_11B_1 4] 1.0 Perfect 
Waco_13_1 19 1.0 Perfect 

















Figure 7 shows a graphical representation of the Table 12: 


Initial Model: Kappa Score Compared to Number of Utterances in Transcript 


Fair Agreement 


Slight Agreement 


1000 1500 
Number of Utterances 





Figure 7. | Kappa Score vs. Number of Utterances in Transcript (Initial Model) 
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The figure suggests that for the initial model and this particular group of 
annotators, there seemed to be an inverse relationship between the kappa score and the 
number of utterances in a transcript. The two transcripts that annotators scored perfectly, 
Waco_11B_1 and Waco_13_1, contained 41 and 19 utterances, respectively. These were 
the shortest transcripts of the entire sample. However, as the number of utterances 
increased, the kappa score decreased significantly. When the utterance level increased 
above 260, the kappa score was always below 0.4, meaning only fair agreement. 
Annotators were not exactly sure of the reason for the correlation between a low number 
of utterances and a high kappa score, but one theory suggested that a low number of 
utterances means a very small number of persuasive utterances. Table 13 shows the 
breakdown of persuasive utterances between transcripts tagged by multiple annotators. 
The parameters of the table are: 

n = the total number of utterances in the transcript 

P, = the total number of utterances marked persuasive by the Annotator A 
Pg = the total number of utterances marked persuasive by the Annotator B 
Pc = the total number of utterances marked persuasive by the Annotator C 


Py = total number of utterances marked persuasive by either annotator 


Table 13. Percent of Transcript Marked as Persuasive (Initial Model) 
































Pa % Pp % Pc % 

Transcript N Pa of n Pp of n Pc of n Pr Pr % of n 
Taylorl 241 12 5.0% | 9 3.7% 17 71% 
Waco_lIA 3 | 134 8 6.0% | 10] 7.5% 14 10.4% 
Waco_12B 2 | 254 5 2.0% 5 2.0% 7 2.8% 
RoganF 562 61 10.9% | 91 | 16.2% 117 20.8% 
Waco_14 925 12} 1.3% 13 | 14% 20 2.2% 
RoganA 269 18) 6.7% 31 | 11.5% 38 14.1% 
RoganB 2122 | 244 | 11.5% 192 | 9.0% 341 16.1% 
Waco_l11B_1}| 42 1 2.3% 1 2.3% 1 2.3% 
Waco_13_1 19 2 10.5% 2 | 10.5% 2 10.5% 
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Figure 8 shows a histogram of the breakdown of percentages in the Table 13. 





Individual Annotator's Percent of Persuasive Utterances 
Compared to Total Utterances (by Transcript) 
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Figure 8. Histogram of Percentage Breakdown of Persuasive Utterances 
in a Transcript 


As seen in Figure 8, the total number of persuasive utterances was usually under 
five percent and never above twenty percent. This correlation means that most of a 
transcript 1s non-persuasive utterances. A smaller number of utterances in the transcript 
implies there will be less persuasive utterances to tag. If there are only one or two 
persuasive utterances in a short transcript (as is the case with Waco_11B_1 and 
Waco_13_1), and the persuasive utterances are obvious to the annotators, then the kappa 
scores will be high. A longer transcript means more chances for disagreement and a 
lower kappa score. Even with the high kappa scores, this overall level of agreement was 
not suitable, because the code book described in Chapter III should be applicable to any 
transcript length. 


se 


C. RETHINKING THE MODEL 


As discussed in Chapter III, annotators went back and discussed the model while 
tagging a 600 utterance transcript together. They tagged the first 300 utterances together 
while discussing the model. After refining the definitions of each category, dropping the 
“reason” category, and adding the “other” category as a catch-all for any persuasive 
utterance that did not fall under the Cialdini principles, the annotators tagged the last 300 
transcripts individually and compared results. The results showed significant 
improvement and are detailed in the following tables. 


Table 14. | Revised Model Results for Second Half of RoganD (Test Transcript) for 
Annotator A and Annotator B 























Annotator B 
Transcript: 
RoganD (last half) Persuasive Non-Persuasive 
Persuasive 
Annotator A 15 f) 
Non-Persuasive ll 106 
pees 0.643 














Table 15. | Revised Model Results for Second Half of RoganD (Test Transcript) for 
Annotator B and Annotator C 


























Annotator B 
Transcript: 
RoganD (last half) Persuasive Non-Persuasive 
Persuasive 
Annotator C 19 3 
Non-Persuasive 7 105 
Kappa 0.747 
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Table 16. Revised model results for second half of RoganD (Test transcript) for 
Annotator A and Annotator C 


























Annotator C 
Transcript: 
RoganD (last half) Persuasive Non-Persuasive 
Persuasive 
Annotator A 15 9) 
Non-Persuasive 4 110 
AADDa 0.731 














The results in the Tables 14, 15 and 16 include the “other” category in the 
calculations. The initial hypothesis of the annotators was that the exclusion of the 
“other” category would increase the level of agreement, because the category was open- 
ended. Any utterance that an annotator felt was meant to be persuasive but was not part 
of the Cialdini categories could be marked with “other.” To test this hypothesis, the 
results were recalculated excluding the “other” category. Annotators were surprised that 
eliminating the “other” category from the results actually decreased the overall agreement 


as shown in Table 17. 


Table 17. | Kappa Score Comparison for Test Transcript with and without "Other" 
Category 





Kappa Score Kappa Score 
Annotators Change 
with “Other” | without “Other” 


























A,B 0.643 0.550 -14.5% 
B,C 0.747 0.453 -39.3% 
A,C 0.731 0.652 -10.8% 





The most likely reason for the increase in agreement using the “other” category is 
because the category has clearly defined parameters in the codebook. Essentially, it is the 
complement of the other seven persuasion categories for any utterance that can be 


marked persuasive. While the category is open-ended, annotators felt that when they 
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defined the category, they covered most of the possible examples where it could be 
applied. This observation suggests that the Cialdini categories do not fully cover the 
space of persuasion categories, at least with respect to hostage negotiations. Given that 
Cialdini’s model is more applicable for business and social models which have some 
substantial differences compared to hostage situations, the model described in our 


research may be more applicable across different domains. 
D. REVISED MODEL RESULTS 


During the annotation of transcripts using the revised model, Annotator C was 
unable to complete the task, so the calculated results are only between Annotator A and 
Annotator B. Tables 18 through 21 show the results between four transcripts that 


included the “other” category. 





























Table 18. Results for Taylor6 Transcript (Revised Model) 
Annotator B 
Transcript: 
Taylor6 Persuasive Non-Persuasive 
Persuasive 1 
Annotator A 36 35 
Non-Persuasive 64 1858 
Reppe 0.707 


























Table 19. — Results for Waco_11B_3 Transcript (Revised Model) 
Annotator B 
Transcript: 
Waco_11B 3 Persuasive Non-Persuasive 
Persuasive 
Annotator A 6 0 
Non-Persuasive ) 220 
pappe 0.853 
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Table 20. Results for Waco_13_2 Transcript (Revised Model) 
Annotator B 
Transcript: 
Waco_13 2 Persuasive Non-Persuasive 
Persuasive 
Annotator A 12 4 
Non-Persuasive 5 291 
mappe 0.713 
Table 21. Results for Waco_16_1 Transcript (Revised Model) 
Annotator B 
Transcript: 
Waco_16_1 Persuasive Non-Persuasive 
Persuasive 
Annotator A ; 3 1 
Non-Persuasive 0 114 
GEDA 0.855 














Table 22 summarizes the agreement between Annotator A and Annotator B for the four 

















transcripts. 

Table 22. ‘Transcript Kappa Scores and Agreement (Revised Model) 
TRANSCRIPT UTTERANCES KAPPA AGREEMENT 
Taylor6 2093 0.707 Substantial 
Waco_11B_3 228 0.853 Almost Perfect 
Waco_13_2 312 0.712 Substantial 
Waco_16_1 218 0.855 Almost Perfect 














For a comparison, Figure 9 shows the agreement between transcripts tagged by 


Annotator A and Annotator B for the original and revised models. It is worth noting that 


the annotators tagged different transcripts in the revised model than in the original one. 
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This adjustment was done to ensure that annotators would not be biased by annotating the 


same transcript again and covering the same material. 















Agreement Between Annotator A and Annotator B 
1 
Almost Perfect Agreement 
0.8 
0.6 - 
& Moderate Agreement ~©- Original Model 
z t= Revised Model 
04 4 7 
Fair Agreement 
0.2 4 
Slight Agreement 
ie) T T T 1 
ie) 500 1000 1500 2000 2500 
Number of Utterances In Transcript 
Figure 9. | Kappa Score vs. Number of Utterances in Transcript (Revised Model) 


This figure shows substantial improvement in the agreement results. In the 
original model, there was some cases of substantial agreement (above 0.6), but only when 
the number of utterances in the transcript was less than 260. In the revised model, there 
is consistent substantial agreement, even when the number of utterances in the transcript 


exceeds 2000. 


Based on the pilot study of the new model where the inclusion of the “other” 
category increased the kappa score, annotators expected a decrease when the “other” 
category was removed from the results. Table 23 shows the change in kappa score when 


the “other” category was removed. 
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Table 23. Kappa Score Comparison With and Without “Other” Category 
(Revised Model) 
Number of | Kappa with | Kappa without 

Transcript Utterances | "Other" "Other" Change 
Taylor6 2093 0.707 0.665 -6.0% 
Waco_11B_3 228 0.853 0.497 -41.8% 
Waco_13_2 312 0.712 0.552 -22.5% 
Waco_16_1 218 0.855 1.000 17.0% 

















comparison of the Tables 24 and 25. 


The last result for Waco_16_1 was unexpected as the kappa score increased when the 


“other” category was removed. An explanation for this result can is provided in the 




















Table 24. | Waco_16_1 Agreement Matrix With “Other” Category 
Annotator B 
Transcript with “Other”: 
Waco_16_1 Persuasive Non-Persuasive 
Persuasive 
Annotator A : 3 1 
Non-Persuasive 0 14 
pee 0.855 


Table 25. 














Waco_16_1 Agreement Matrix Without “Other” Category 





Transcript without “Other”: 


Annotator B 




















In the first table, there were only four utterances in the entire transcript that either 








Waco_16_1 Persuasive Non-Persuasive 
Persuasive 
Annotator A ; 1 0 
Non-Persuasive 0 017 
pappa 1.00 
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Annotator A or Annotator B marked as persuasive, about 1.8 percent of the transcript. In 





the second table, the number of persuasive utterances was reduced to just one, about 0.4 
percent of the transcript. In other words, the persuasive utterances marked with “other” 
by either annotator comprised 75 percent of the total persuasive utterances in the 
Waco_16_1 transcript. When the “other” category was removed, there was only one 
utterance that the annotators tagged as persuasive. Moreover, the number of persuasive 


utterances is still sparse as show in the Table 26 (labels correspond to those in Table 13). 


Table 26. Percent of Transcript Marked as Persuasive (Revised Model) 



































Pa percent Pg percent Pr percent 
Transcript N Pa of n Pp of n Pr of n 
Taylor6 2093 | 171 8.2% 200 9.6% 235 11.2% 
Waco_11B_3 | 228 6 2.6% 8 3.5% 8 3.5% 
Waco_13_2 | 312 16 5.1% Ly 5.4% 21 6.7% 
Waco_16_1 218 4 1.8% 3 1.4% 4 1.8% 











The positive change shown for Waco_16_1 in Table 23 can be attributed to the 
sparseness of persuasive data as well as the large proportion of utterances tagged with 
“other” compared to the other seven persuasion categories. 

For reference, a fully tagged transcript is provided in the appendix. The transcript 
(Waco_11B_1) is relatively short (41 utterances) but it will give the reader insight into 


the experiment and show examples of what the annotators did for each transcript. 
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V. CONCLUSION AND FUTURE WORK 


In this research, we developed a model for tagging persuasion in conversation 
based on the social-psychological model of James Cialdini. Cialdini’s persuasion 
categories were reciprocity, commitment and consistency, liking, scarcity, authority and 
social proof. We also added “reason” as a category for our model. Using this model, 
annotators tagged the utterances of 37 hostage negotiation transcripts as persuasive or 
non-persuasive and compared the results using Cohen’s Kappa score. Based on 
disappointing initial results, we eliminated the “reason” category and added “other” as a 
catchall for anything not covered by Cialdini’s categories. The experiment was rerun 
using the revised model and the results showed significant improvement in agreement 
between the annotators and formed the basis for an adjudicated corpus of persuasive 


conversations. 
A. CONCLUSION 


Based on these results, the annotators concluded that the revised model for 
tagging persuasion is superior to the original one. This improvement is most likely due to 
increased structure in the revised model with more specific guidelines for annotators 
when tagging persuasion. The more specific criteria covered the Cialdini principles but 
also included adding the “other” category. From conversations with annotators, they 
were also able to more accurately tag the data with better experience. In that respect, the 
initial model was good practice for the annotators because many of the same types of 
persuasive utterances were tagged again in the revised model. It is also worth noting that 
hostage negotiations are unique circumstances filled with stressful situations and 
determined individuals. Although there are some similarities with other negotiations 
(business, political, etc.), hostage negotiations are fundamentally different. Life and 
death situations have a way of galvanizing individuals and may introduce methods of 


persuasion not covered by Cialdini. 


Even though the results from the revised model showed substantial agreement, 


annotators felt that in order to perform any machine-learning on the data, a final 
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adjudicated data set had to be completed. Based on this need, Annotator A and 
Annotator B went through the entire data set and tagged each transcript together to 
produce a final persuasion corpus. The final corpus will hold each of the 37 transcripts 
with eight persuasion categories for each utterance in a transcript. If annotators marked 
more than one category in an utterance, they ranked the numbers in accordance of 
importance (highest to lowest). This corpus will be used in future work with computer 
science for persuasion-detection in conversation and provide a basis for machine-learning 


techniques to automatically separate persuasive and non-persuasive utterances. 
B. FUTURE WORK 


With an adjudicated persuasion corpus, there is potential for developing machine- 
learning techniques to automatically divide the data into persuasive and non-persuasive 
utterances. Computational models such as Naive Bayes and Support Vector Machines 
could be useful in accomplishing this goal. If these models prove successful, they might 
further be applied to a non-annotated corpus such as those from the perverted-justice 
corpus (Perverted-Justice.com). While the perverted-justice corpus is internet chat-based 
and not conversation transcriptions such as the ones in our research, there may be some 


techniques developed on our corpus that are transferrable to other domains. 


Dialogue tagging could be another useful avenue of study with the adjudicated 
corpus. Each utterance would have to be broken down into separate sentences. Some of 
the transcripts contained punctuation and others were simply the transcription of the 
words of the conversation without punctuation, so the corpus would need to be 
standardized first. Once the sentences are tagged with dialogue acts, researchers could 
look for any correlation between particular dialogue tags (or sequences of dialogue tags) 


and persuasion attempts. 


Additional work needs to be done on how persuasion is organized in 
conversation. Figure 10 shows the agreement between annotators during the testing of 
the new model (from Figure 6 in Chapter III). While tagging the data, annotators noticed 
that the hostage taker and negotiator would often have long spells of non-persuasive 


utterances. This is not unexpected because, as noted in Chapter IV, conversations are 
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mostly non-persuasive utterances. The annotators did notice that although the persuasive 
utterances in a transcript were sparse, there were many sections with persuasive 
utterances close together as seen in Figure 10. In other words, there were long spells of 


non-persuasive utterances followed by a few persuasive utterances. 


Persuasive 


New Model: Individually Tagged Utterances 
ae = y lagg 


‘ 





Not 
Persuasive 


Figure 10. Persuasion Sections Based on Agreement Between Annotators for 
Individually tagged Utterances for the Test Transcript in the Re-designed Model 


Figure 11 is a histogram of Annotator A’s tagging results from the re-designed 


model that shows the number of utterances between persuasive utterances by speaker. 
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Annotator A: Number of Utterances Between Persuasion by Speaker 
(Pilot Study for Re-designed Model) 


22 


Frequency 








| A 4 P 4 
ie [J Es] 
oO 
o | _ = | -_ = _ = 
2 bs | 5 10 20 50 More 








Figure 11. Histogram of the Number of Utterances Between Persuasive Utterances 
(by Speaker) for Annotator A in the Test Transcript 


Most persuasive utterances only had one utterance separating them, and this one 
utterance was usually the other party’s response to the persuasion attempt. Intuitively, 
the negotiator usually tried to build some kind of rapport with the hostage taker by 
listening to his or her demands, discussing logistics and other issues, and then tried a 
persuasion technique. This technique can take several utterances to develop, like the 
rejection-then-retreat approach of reciprocity discussed in Chapter II. It would be useful 
to use the adjudicated corpus and perform the same kind of analysis in coordination with 
any machine-learning systems developed for the corpus. If the system has difficulty 
recognizing individual utterances as persuasive, it may have more success determining if 


certain sections of the conversation are persuasive. 


There are other methods used in perspective analysis that may be applied to 
persuasion detection. Some persuasion is the negotiator attempting to show that his or 
her point of view is the correct one, and that the hostage taker should surrender. While it 
may be trivial to determine which perspective the negotiator and hostage taker have, it 
may be useful to use subjective sentence classifiers (Riloff & Wiebe, 2003) to determine 


if the utterance is subjective and if that subjectivity correlates to any persuasion attempt. 
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Additionally, it may be useful to refine the “other” category, described at the end 
of Chapter III, into separate cases. A category like “duty” may be appropriate for 
utterances that deal with family members. Family members like a hostage taker’s 
children or younger siblings may not be authority figures but can still influence his or her 
decisions. Other categories may be “self-consistency” or “rapport building.” Some of 
these cases may even be extensions of the Cialdini categories that annotators did not 
recognize, but may be useful to split the cases of the “other” category into separate 


instances. 


Finally, while not all persuasive utterances contained an extensive number of 
words, annotators did recognize that the longer the negotiator spoke in an utterance, the 
more likely that utterance contained some form of persuasion. To explore this 
observation, a transcript was randomly selected (Waco_12A_2) from the adjudicated data 
set and was divided into persuasive and non-persuasive utterances. We compared the 
total number of each kind of utterance versus the average text length (the number of 
characters). Figure 12 shows that non-persuasive utterances were much more prevalent in 
the Waco_12A_2 transcript, but Figure 13 shows that persuasive utterances had a larger 


average text length than non-persuasive utterances. 
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Utterances Type Comparisonin Waco_12A_2 


Number of Utterances 


Persuasive Non-Persuasive 





Figure 12. Count of Utterance Types in Waco_12A_2 


Average Text Length of Utterance Type in Waco_12A_2 
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Figure 13. Average Text Length (in Characters) of Persuasive and Non-Persuasive 
Utterances in Waco_12A_2 
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As the figures show, while non-persuasive utterances are more prevalent 
throughout the transcript, the average persuasive utterances is 236% longer than the 
average non-persuasive utterance. This kind of attribute should be explored on the entire 


adjudicated corpus and perhaps added as a feature-vector in a machine-learning system. 
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APPENDIX 


The table is an example of a full transcript with annotation. The transcript is from 
the Waco tapes (Texas State University, 1993) and was annotated by Annotator B. In the 
transcript, David is the leader of the Branch Davidians, Rita is one of David’s followers, 


and Henry is an FBI negotiator. 


ANNOTATOR B 





Utterance OTHER 





HENRY: Rita? 








HENRY: Hello? 








HENRY: How are you 
doing? 











HENRY: You making 
rogress? 





HENRY: Well, I'm 
inside of a room. I can't 
really -- I don't have a 
window. 


























HENRY: Is that right? 
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Utterance 


HENRY: About - 


HENRY: In about 15 
minutes? 


IHENRY: Okay. You -- 
and we're still on-line 
for the rest of the stuff. 
That is I mean we still 
Ihave the two women 
and, and two, and two 
Kids that will -- 


HENRY: Okay. Okay. 


IHENRY: Okay. 


IHENRY: Okay. Very 
good. 


HENRY: Okay. Now try 
to keep it short because, 
you know, the, these 
people are not going to 
want to play something 
that's real long, okay? 


OTHER 


ANNOTATOR B 















































IHENRY: Okay. 
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Utterance 


OTHER 


ANNOTATOR B 





HENRY: Well, you 
know, remember _ the, 
the thing that you did 
for Jim before? 


HENRY: About half a 
page or so. You know, 
ou -- 


IHENRY: You, you don't 
Ihave to make it that 
short. But you know -- 


HENRY: But, but don't 
make it like a half hour 
or something like that. 


IHENRY: Okay. 
IHENRY: Okay. 


HENRY: I'll talk to you 






































later. Bye. 
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